The Purposed Model¶
As we discussed in the previous part, the purposed model should has the ability to capture the relationship within features, especailly within the time-series features. Besides, the purposed model should have robustness to noise. So, we will design a neural network model which contains TCN layer and self-attention layer as the purposed model.
In theory, the TCN can use convolutional layers to complete feature extraction and use dilated convolution layers to capture the relathionship and within the input information and recognise the hidden patterns in the interaction components. But the noise may increase when the TCN produces interaction components (or new features) via feature extraction.
So, we use the self-attention layer to help our model to focus on important interaction components and information, thus enhancing our model robustness to noise.
The introduction of attention and TCN are as follows:
（1）Temporal Convolutional Networks (TCN)¶
TCN refers to a class of network structures which apllying convolutions in time-series analysis. In our model, the TCN structure we use is:
TCN = 1D Fully Convolutional Network (FCN) + causal convolutions.
FCN can extract features and combination features. And the causal convolutions is a kind of multiple-layer convolution of temporal data, which ables models learn information form time-series data and ensures that the model does not violate the order in the time-series data.
Compared with RNN or LSTM, the TCN have more powerful potential to reveal hidden pattern from the time-series data and capture relationships within the features due to the FCN and causal convolutions. As for our TCN model, is could be regards as a combination of convolution and RNN.
（2）Self-Attention¶
Self-Attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence. In other words, self-attention mimics the human attention mechanism, allowing the model to focus on learning important information and ignore unimportant information. More specifically, the self-attention can be expressed mathmatically as follows:
For input dataset $X \in R^{n \times m}$, where $n$ is the number of samples and $m$ is the dataset dimension.
The self-attention mechanism will create three weight matrices $W^Q$, $W^K$, $W^V$ according to the input samples, and then generate three new matrices $Q$ (query), $K$(key ), $V$(value), where
$$ Q = X W^Q, \text{ where } W^Q \in \R^{m \times m_q} $$$$ K = X W^K, \text{ where } W^K \in \R^{m \times m_k} $$$$ V = X W^V, \text{ where } W^V \in \R^{m \times m_v} $$
Acturally, the names of "query", "key" and “value” come from the recommendation system. The basic idea is: given a query, calculate the correlation between the query and the key, and then find the most appropriate value according to the correlation between the query and the key. One case is movie recommendations. The query is a person's preference information for movies (such as point of interest, age, gender, etc.), the key is the type of the movie (comedy, era, etc.), and the value is the movie to be recommended. In this case, although each attribute of query, key and value is in a different space, they actually have a certain potential relationship, that is to say, through some transformation and calcalting the correlation between the key and the query, we can find out the appropriate value (movie recommended).
Next, the self-attention mechanism will use $Q$ and $K$ to calculate the similarity and standardize the scale:
$$Similarity = \text{softmax}(\frac{QK^T}{\sqrt{m_v}})$$
Finally, the similarity will be use as weights for calulating the Attention:
$$\text{Attention}(Q,K,V) = softmax(\frac{QK^T}{\sqrt{m_v}})V$$
As we can see, in conclusion, the essence of the self-Attention function can be described as a mapping from a query to a series of (key-value-value) pairs.
Self-Attention can do feature extraction and help our model focus on important information. So, the self-Attention can improve the performance of the model and the robustness to noise.
（3）Random Forest (the benchmark model)¶
We choose Random Forest as the benchmark model. Because Random Forest can capture the relationship features. Besides, Random Forest can handle unbalanced datasets.
However, Random Forest still has drawbacks. It may be affected by noise and then be easy to overfit on classification. And time-series data often contain a lot of noise. Therefore, in theory, the Random Forest model can achieve good performance, but there is still room for improvement.
If our purpose model can identify hidden patterns and relationships within the input features, and have a robustness to noise. Then it will likely outperform Random Forest in performance.
（4）The purposed model (the TCN-Attention neural network model)¶
The model divides the features into two parts: time series features and non-time series features. Learn from the idea of encoding and decoding in transformer, which enalable models to learn information from context or relationships whitin features. Our purposed model will implement the encoding of the two parts of features separately, and then combine the two parts of the encoded content togher. At last, the model will decode the encoded content and get the prediction of the probability of default.
For time-series features, we firstly use the TCN layer to extract features to capture the relathions within features, and then use the self-attention layer to let model can pay more attention to important information and be less disturbed by noise. And then, we will use a Feedforward neural network to achieve the combination of features. Now, the model can get the enconded content of time-series features.
For non-time series features, the model will use a feedforward neural network to splite and combine the information to finish encoding.
Then the model uses the Concatenate layer in Keras to stitch the two parts of the data together. Next, the model implements decoding with a self-attention layer. This allows the model to capture the relationships between time-series features and constant features. At last, through a feedforward neural network which contains a single output neuron with softmax activation function in the output layer. The model can finally predict the default probability.
Besides, in order to overcome the overfitting problem, we will use the elu activation function in the midden layers. And we will add some dropout layers(which can randomly drop some information) in our model.
More specifically, the model structure is shown in Figure 5.
